High-Efficiency Video Coding (HEVC) is a new international video coding standard that is being developed by the Joint Collaborative Team on Video Coding (JCT-VC). HEVC is based on the hybrid block-based motion-compensated DCT-like transform coding architecture. The basic unit for compression, termed Coding Unit (CU), is a 2N×2N square block, and each CU can be recursively split into four smaller CUs until a predefined minimum size is reached. Each CU contains one or several variable-block-sized Prediction Unit(s) (PUs) and Transform Unit(s) (TUs). For each PU, either intra-picture or inter-picture prediction is selected. Each TU is processed by a spatial block transform and the transform coefficients for the TU are then quantized. The smallest TU size allowed for HEVC is 4×4.
The quantization of transform coefficients plays an important role in bitrate and quality control in video coding. A set of quantization steps is used to quantize the transform coefficient into a quantization level. A larger quantization step size will result in lower bitrate and lower quality. On the other hand, a smaller quantization step size will result in higher bitrate and higher quality. A straightforward implementation of the quantization process would involve a division operation which is more complex in hardware-based implementation and consumes more computational resource in software-based implementation. Accordingly, various techniques have been developed in the field for division-free quantization process. In HEVC Test Model Revision 5 (HM-5.0), the quantization process is described as follows. A set of parameters are defined:
B=bit width or bit depth of the input source video,
DB=B−8,
N=transform size of the transform unit (TU),
M=log 2(N),
Q[x]=f(x), where f(x)={26214,23302,20560,18396,16384,14564}, x=0, . . . , 5, and
IQ[x]=g(x), where g(x)={40,45,51,57,64,72}, x=0, . . . , 5.
Q[x] and IQ[x] are called quantization step and dequantization step respectively. The quantization process is performed according to:qlevel=(coeff*Q[QP %6]+offset)>>(21+QP/6−M−DB), whereoffset=1<<(20+QP/6−M−DB),  (1)where “%” is the modulo operator. The dequantization process is performed according to:coeffQ=((qlevel*IQ[QP%6]<<(QP/6))+offset)>>(M−1+DB), whereoffset=1<<(M−2+DB).  (2)
The variable qlevel in equations (1) and (2) represents the quantization level of a transform coefficient. The variable coeffQ in equation (2) represents the dequantized transform coefficient. IQ[x] indicates de-quantization step (also called de-quantization step size) and QP represents the quantization parameter. “QP 16” in equations (1) and (2) represents the integer part of QP divided by 6. As shown in equations (1) and (2), the quantization and dequantization processes are implemented by integer multiplication followed by arithmetic shift(s). An offset value is added in both equations (1) and (2) to implement integer conversion using rounding.
The bit depth of the quantization level is 16 bits (including 1 bit for sign) for HEVC. In other words, the quantization level is represented in 2 bytes or a 16-bit word. Since IQ(x)<=72 and QP<=51, the dynamic range of IQ[x] is 7 bits and the “<<(QP/6)” operation performs left arithmetic shift up to 8 bits. Accordingly, the dynamic range of de-quantized transform coefficient coeffQ, i.e., “(qlevel*IQ[QP %6])<<(QP/6)”, is 31 (16+7+8) bits. Therefore, the de-quantization process as described by equation (2) will never cause overflow since the de-quantization process uses 32-bit data representation.
However, when quantization matrix is introduced, the de-quantization process is modified as shown in equations (3) through (5):iShift=M−1+DB+4.  (3)if(iShift>QP/6),coeffQ[i][j]=(qlevel[i][j]*W[i][j]*IQ[QP%6]+offset)>>(iShift−QP/6), whereoffset=1<<(iShift−QP/6−1), with i=0 . . . nW−1,j=0 . . . nH−1  (4)elsecoeffQ[i][j]=(qlevel[i][j]*W[i][j]*IQ[QP%6])<<(QP/6−iShift)  (5)
wherein “[i][j]” indicates the position (also called indices) of the transformed coefficient within a transform unit, W denotes quantization matrix, nW and nH are width and height of the transform. If n represents the dynamic range of a quantization level for a transform coefficient, the dynamic range n has to satisfy the following condition to avoid overflow:n+w+iq+QP/6−M+DB−3≦32,  (6)
where w is the dynamic range of quantization matrix W, iq is the dynamic range of IQ[x] and the bit depth of the de-quantized or reconstructed transform coefficient is 32 bits.
If the dynamic range of the quantization matrix W is 8 bits, the dynamic range of the reconstructed transform coefficient as described by equations (3) through (5) becomes 34 (16+8+7+3) bits for QP=51, M=2 and DB=0. When the de-quantization process uses 32-bit data representation, the reconstructed transform coefficient according to equation equations (3) through (5) may overflow and cause system failure. Therefore it is desirable to develop a scheme for transform coefficient reconstruction to avoid possible overflow.